In redundant control systems, such as redundant control cards, synchronicity between the control systems is maintained by means of transaction messages. When the state of the system being controlled, such as a communications network, changes, software modules on the active control system are triggered to update stored state information. The software module also sends a transaction message to the standby control system, and in response the standby control system updates its version of the stored state information. For example, if a channel is created, a software module is called to populate the channel table entry. The software module also sends a transaction message to the standby control system to indicate the channel table entry change. The standby control system responds to this transaction message by populating its own channel table entry. In this way, when a switchover occurs, the newly active control system will have up to date information which accurately reflects the state of a system being controlled.
In systems having state dependencies, an event may trigger multiple transaction messages. In such systems, an event triggers a first software module to update state information and send a transaction message to the standby control system. The first software module also triggers a second software module responsible for maintaining state information which is dependent on the state information maintained by the first software module. The second software module updates the state information on the active control system, then sends its own transaction message to the standby control system. Continuing the example given above, once the first software module (a channel module) has populated the channel table entry, it may trigger the second software module (a forwarding module) to allocate a forwarding index for the new channel. The first software module and the second software module each send their own transaction message to the standby control system. Of course, there may be more than two levels of dependency, and the second software module may trigger a third software module with its own transaction message, and so on.
A problem can arise if a switchover occurs while transaction messages are being sent. The standby control system would then have incomplete state information. This can lead to various problems, such as incomplete state information or leakage of resources. For example, if the channel module above sent its transaction message but the transaction message of the forwarding module was not sent in time (i.e. before the active control system failed), then the standby control system (now the newly active control system) will act as if the channel has been created and may assume that the forwarding index has been allocated. It is only when use of the new channel is attempted will the absence of information become known and an error occur. If on the other hand it was the channel module that was unable to send its transaction message in time, then the standby control system will have processed the transaction message from the forwarding module and will have allocated a forwarding index for a non-existent channel. When the channel is created anew on the newly active control system, the channel module on the newly active control system will trigger a forwarding module and a new forwarding index will be allocated. This results in leakage of forwarding indices.
To overcome this risk, intra-module audits are carried out following a switchover. An intra-module audit examines the state information of a software module against each of its dependent software modules to determine whether all the interdependent state information is present, and to allocate or deallocate resources in an attempt to correct any deficiencies. However, intra-module audits can be extremely complex, since the dependencies between software modules are generally not as simple as in the example given above. Compounding the problem is that this complexity is called upon just after a switchover when there is no additional standby control system. The complexity is also hard to maintain and debug.